Deep manifold-to-manifold transforming network for action recognition

In this paper, a novel deep manifold-to-manifold transforming network (DMT-Net) is proposed for action recognition, in which symmetric positive definite (SPD) matrix is adopted to describe the spatial-temporal information of action feature vectors. Since each SPD matrix is a point of the Riemannian manifold space, the proposed DMT-Net aims to learn more discriminative feature by hierarchically transforming the data points from one Riemannian manifold to another more discriminative one. To this end, several novel layers are proposed in DMTNet, including SPD convolutional layer, channel convolution layer, diagonalizing layer and kernel regression layer. Specifically, SPD convolutional layer enables multi-channel convolution to be well applied to Riemannian manifold, and the kernel regression layer enables the distance metric computation between two SPD points in Riemannian manifold to be done in the Euclidean space, in which a novel reference set dynamically generated during the network training is also introduced to relieve the computational burden of the kernel method. To evaluate the effectiveness of the proposed method, three action recognition databases are respectively used to testify our method and the experimental results show that our algorithm outperforms the state-of-the-art methods.